Dorling et al. Health Research Policy and Systems 2014, 12:12 -^jf^ 

http://www.health-policy-systems.eom/content/12/1/12 i^4f W HEALTH RESEARCH POLICY 

" AND SYSTEMS 



RESEARCH Open Access 



Developing a checklist for research proposals to 
help describe health service interventions in UK 
research programmes: a mixed methods study 

Hannah Dorling" Donna White, Sheila Turner, Kevin Campbell and Tara Lamont 



Abstract 

Background: One of the most common reasons for rejecting research proposals in the National Institute for Health 
Research (NIHR) Health Services and Delivery Research (HS&DR) Programme is the failure to adequately specify the 
intervention or context in research proposals. Examples of failed research proposals include projects to assess 
integrated care models, use of generic caseworkers, or new specialist nurse services. These are all important service 
developments which need evaluation, but the lack of clarity about the intervention and context prevented these 
research proposals from obtaining funding. The purpose of the research presented herein was to develop a 
checklist, with key service intervention and contextual features, for use by applicants to the NIHR HS&DR 
Programme to potentially enhance the quality of research proposals. 

Methods: The study used mixed methods to identify the need for and develop and test a checklist. Firstly, this 
included assessing existing checklists in peer-reviewed literature relevant to organisational health research. Building 
on existing work, a new checklist was piloted. Two reviewers used a small sample (n = 16) of research proposals to 
independently assess the relevance of the checklist to the proposal and the degree of overlap or gaps between the 
constructs. The next two stages externally validated the revised checklist by collecting qualitative feedback from 
researchers and experts in the field. 

Results: The initial checklist was developed from existing checklists which included domains of intervention and 
context. The constructs and background to each were developed through review of existing literature. Eight 
researchers provided feedback on the checklist, which was generally positive. This iterative process resulted in 
changes to the checklist, collapsing two constructs and providing more prompts for others; the final checklist 
includes six constructs. 

Conclusions: Features relating to intervention and context should be well described to increase the quality of 
research proposals and enhance the chances of the research receiving funding. Existing checklists do not have 
enough focus on areas relevant to research proposals in complex health service interventions, such as workforce. 
A formative checklist has been developed, and tested by end users. Tentative findings suggest usefulness and 
acceptability of such a tool but further work is needed for full validation. 
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Background 

The National Institute for Health Research (NIHR) Health 
Services and Delivery Research (HS&DR) Programme 
funds research in the UK for managers and service leaders 
on the quality, access, and organisation of health services, 
including evaluations of how the National Health Service 
(NHS) might improve delivery of services. Additionally, 
the programme aims to support research that is likely to 
lead to changes in practice that will have a significant im- 
pact, is likely to generate new knowledge of direct rele- 
vance to the NHS, and has the potential for findings to be 
applied to other conditions or situations outside the im- 
mediate area of research [1]. When considering research 
proposals in a competitive funding process, weight is given 
to the teams track record and the appropriateness of the 
research design. However, one of the most common rea- 
sons for which the funding board rejects research pro- 
posals is failure to adequately specify the intervention or 
contextual features in research proposals. Many complex 
interventions are staff-based, but little information is given 
on the composition of the team in terms of skill-mix and 
grade-mix. Other factors which may be harder to measure 
include aspects of organisational culture, including levels 
of staff engagement or the degree to which the organisa- 
tion is data-driven or embraces change. Examples of failed 
research proposals include projects to assess new inte- 
grated care models, use of generic caseworkers, specialist 
nurse service, and streamlined services. These are all im- 
portant complex service developments that need evalu- 
ation, but have lacked clarity about the intervention and 
context, which meant that opportunities to obtain funding 
for valuable research were missed. 

Complex interventions are often described as "inter- 
ventions that contain several interacting components, 
but they have other characteristics that evaluators should 
take into account" [2]. The Medical Research Councils 
complex interventions guide notes that the active ingre- 
dient' in healthcare interventions can be challenging to 
describe [3]. This can include components of an inter- 
vention such as behaviours and methods of organising 
and delivering those behaviours, for example different 
types of practitioners, settings, and locations. This is 
unlike simple' interventions, such as a drug where re- 
searchers face fewer problems in providing details where, 
for instance, the intervention can be defined in terms of 
dose and frequency [4], whereas one of the key details 
when evaluating complex interventions is to consider 
whether they are generalizable to everyday practice. 
Therefore, it is essential to understand the whole range 
of potential effects and how they may alter, for example, 
among recipients, staff, or between sites [2]. Such initia- 
tives may be poorly described, such as exact staff and 
grade mix which limits the fidelity and reproducibility of 
the intervention. Many workforce type interventions 



build upon existing services and it may be difficult to 
isolate the particular features of the intervention which 
are new and distinct from care as usual', as required for 
experimental or quasi-experimental design. Other fac- 
tors which may be harder to communicate include as- 
pects of organisational culture, including levels of staff 
engagement or the degree to which the organisation is 
data-driven or embraces change. 

Often in health services research there is a blurring 
between intervention and context [5], particularly in 
workforce interventions, which can increase the diffi- 
culty of designing robust evaluations. A lack of clarity 
could in part be due to the complex nature of contexts; 
therefore, the role of context in intervention develop- 
ment needs to be well understood to reduce ambiguity 
[6]. This blurring between intervention and context may 
make it more challenging when writing research pro- 
posals, due to the multifaceted nature of such interven- 
tions. For instance, important elements of the research 
proposal may be overlooked or undervalued, but need to 
be embedded within the research. A checklist may be 
useful to help researchers identify these elements and 
help explicitly explain how they will be considered in a 
research proposal. 

A checklist can be a useful prompt for completeness of 
key information or processes in any given activity [7]. For 
example, Taylor proposes a checklist with classified fea- 
tures of context which may be particularly relevant when 
evaluating patient safety interventions [6]. Other tools typ- 
ically focus on checklists that aim to improve the report- 
ing of research, such as the CONSORT statement for 
clinical trials [8] and SQUIRE guidelines on implementing 
service improvements [9]. Similarly, other publications 
have presented evidence, systematically reviewing studies 
that evaluate interventions [10]. For instance, checklists 
have been used to ascertain whether elements of the inter- 
vention were fully described. Glasziou et al. reviewed 80 
studies and concluded that over half were missing details 
[11]. Schroter et al. developed a checklist and used it to 
evaluate 51 trials published in the British Medical Journal 
[12]. More recently, Douet et al. used this checklist to as- 
sess whether NIHR Health Technology Assessment rando- 
mised controlled trials published in the Health Technology 
Assessment journal were described in sufficient detail to 
replicate in practice [13]. Much of this work is focused on 
research around therapeutic or diagnostic interventions or 
procedures, which tend to be more defined and well- 
bound than complex people-based service interventions. 
Nevertheless, all the checklists have a common theoretical 
underpinning; they are a mnemonic device, with a list of 
components or constructs in order to perform a certain 
task. This function makes them useful in evaluation, since 
a systematic approach can be used to determine value 
complex factors. However, none of these checklists focus 
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on improving the quality of research proposals, but their 
content and benefits are well recognised and valuable to 
researchers. 

Taking these points into consideration, it was felt that 
the development of a checklist may be a useful tool that 
could enable less experienced applicants to more suc- 
cessfully address the complex set of requirements in- 
volved in writing research proposals to the NIHR 
HS&DR Programme. In turn, it is hoped that such a tool 
may enhance the quality of research proposals, so the 
programme can fund important complex service devel- 
opments which need evaluation. This is not expected to 
solely answer the problem, but begin to improve the 
situation. Therefore, the purpose of our study was to de- 
velop a checklist, with key service intervention and con- 
textual features [14]. 

Methods 

A mixed methods approach was used in four stages to 
meet the aims of the research. 

Stage one 

Firstly, an initial checklist was designed by HD and TL, 
based on a review of existing checklists, reporting stan- 
dards, and relevant literature. A snowball sampling tech- 
nique was used to identify literature, which was prompted 
by reading a patient safety checklist paper [6]. This tech- 
nique allowed us to produce a collection of relevant arti- 
cles built around the initial article and facilitated insights 
into a broad background of research. Literature that was 
considered was firstly informed by what is needed in 
research proposals and is relevant to applicants to the 
programme and, secondly, by an understanding of com- 
plex interventions that are delivered by the NHS and their 
context. 

Stage two 

The second stage was to pilot the checklist, by conducting 
a retrospective review of a sample of previous applications. 
To provide focus, this sample was kept to research pro- 
posals received in response to two NIHR HS&DR invita- 
tions to apply for research funding. Since the checklist 
concentrates on research employing an intervention, 
secondary, theoretical, and methodological studies were 
excluded. From the two invitations for funding, 16 full re- 
search proposals were included, which covered a balance 
of service and workforce interventions across a range of 
health settings and care groups. These research proposals 
described studies with a wide range of settings and inter- 
ventions such as training packages, community healthcare 
services, discharge packages, and hospital design. 

Initially, two reviewers (HD and DW) examined 4 out 
of the 16 research proposals independently, assessing the 
relevance of the checklist to the proposal, and the degree 



of overlap between the constructs. The results of this ini- 
tial investigation were then discussed in person to assess 
the level of agreement between reviewers. The remaining 
research proposals were then reviewed and discussed. The 
two reviewers examined the relevance of the initial check- 
list to the research proposals and made judgements on 
whether the checklist constructs were useful, whether the 
appropriate terminology was used, and whether any ele- 
ments were missing. Disagreements were resolved through 
consensus discussion supervised by TL. 

Stage three 

The next stage was to validate the content of the check- 
list and determine whether it was a suitable tool for re- 
searchers to use. This was done in several steps. The 
first step was to gain feedback from researchers who 
were preparing to submit detailed full proposals to the 
NIHR HS&DR Programme in response to the invitation 
to apply for funding for studies on 'the organisation and 
delivery of 24/7 healthcare'. Out of the five researchers 
contacted, three were able to give feedback. 

Stage four 

To further validate the checklist, the next stage was to 
contact researchers who currently had a NIHR HS&DR 
funded project. A meeting welcoming those working on 
newly funded projects to the NIHR was used as an op- 
portunity to make contact with researchers. Researchers 
from each project were asked to complete the checklist 
for their study (unless they felt their study did not have 
an 'intervention', as the checklist would not be applicable 
to them). Five researchers were able to give feedback, 
which was collected by HD via telephone and email, 
using semi-structured questions. We asked about their 
experience of using the checklist, its usability, and the 
suitability of the content of each checklist item. Lastly, 
the checklist content was assessed by consultant advisors, 
who provide professional advice to the NIHR HS&DR 
Programme and regularly view research proposals submit- 
ted to the Programme. 

Results 

Stage one 

Eight key constructs were identified from the literature re- 
view - organisation, location, patient group, workforce, 
staffing, activity, culture/leadership, and costing (Figure 1). 
An initial checklist was designed, containing a very brief 
description of each of these constructs and specific 
prompts for each. Many of the checklist constructs origi- 
nated from other checklists, such as those developed in 
the research by Taylor [4] and Schroter [10], reporting 
tools [6,7], and other literature on complex interventions 
[15,16] and context [17,18], including important features 
such as leadership [19,20] and culture [21]. 
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Figure 1 The development of the checklist constructs. 



Stage two 

The checklist constructs were adapted for use by re- 
searchers by giving further explanation to clarify the back- 
ground to each one by considering their relevance to 
NIHR HS&DR research proposals. Once this had been 
completed a number of changes were made, which are dis- 
cussed further on. 

Stages three and four 

On the whole, the researchers responded positively to 
the checklist; a sample of the feedback from researchers 
is presented in Table 1. The researchers appreciated hav- 
ing a tool to aid them in writing their research proposals 



and commented that it made them think about factors 
that they would not have considered previously. Overall, 
the essence of the checklist appeared to be clearly 
understood. 

The main construct which researchers had not consid- 
ered when writing research proposals was the other con- 
textual information'. This may be due to its complex 
nature, unlike a simple intervention. Some of the principal 
investigators for service evaluations were clinicians without 
experience in carrying out complex mixed-methods evalu- 
ations. The inclusion of this general construct appeared 
helpful in prompting researchers to consider issues such as 
sampling strategy ("how 'typical' is my organisation?") and 
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Table 1 Perceptions of the checklist 



Positive aspects 



Concerns and suggestions for 
improvement 



"It gives a good visual cue" 

"It gives you the confidence to write a good proposal as it is a good starting point" 
'The checklist reinforced the importance of the big picture rather than a selective focus" 
"It is a very useful tool and has a good outline and structure" 

'The checklist enables you to be more explicit about what to write in research proposals and it clarifies the 
question" 

'The main question summarises the important core to our research proposals" 

"It is very applicable to the current proposal currently being worked on, especially as there is lots of patients 
and workforce information" 

"I would certainly read over to match this with the research proposals... and would download it off the 
website" 

"We used it in our meeting to write the full proposal... the team really liked it!" 

'The checklist was helpful, particularly in thinking through the many and varied aspects of context" 

"Depending on the proposal content, for instance if it was an intervention study, I would access and use it" 

'The most useful box was the contextual information, not many of those details would have been added if the 
box was not there" 

"A weighting might be useful" 

"Another box which could be added is data sources, for instance who collected it, is there anything unique and 
the measurement of the outcomes" 

"Maybe it should be used at the initial outline proposal stages?" 

'To improve the checklist need to include points on which other groups are likely to be affected by the change 
in service delivery" 

"If the language used was more generic i.e., not focussed towards intervention studies it could be more widely 
employed" 

'The checklist should fit on one page, anything bigger than that would probably put people off from reading 
it" 

"It would take too long to fill out" 



attribution ("how will I know that the results are not due 
to the charismatic medical director in that hospital?") in 
the research design. 

Some researchers also had concerns about the check- 
list and recommendations for improvements were given. 
For instance a couple of the researchers interpreted the 
meaning of a few words differently to what had been 
intended, therefore, the wording was changed to im- 
prove clarity. Other recommendations were made, such 
as giving a weighting to each category, but this was not 
applicable to meet the aims the checklist, therefore, no 
further changes were made. 

Consultant advisors from the NIHR HS&DR Programme 
fed back that some of the wording needed to be updated 
due a recent reorganisation within the UK's NHS, but 
otherwise they felt the checklist was suitable for re- 
searchers to use to make a good quality research proposal. 

The final checklist includes six constructs, each with a 
few bullet points to the background (Table 2). Decisions 
about whether to include or exclude each construct were 
made during the different stages of the project, as illus- 
trated in Figure 1. 



Organisation 

Organisation is the first construct in the checklist. It is 
important to understand the characteristics of an organ- 
isation, as it can have an influence on how an interven- 
tion is delivered and its effectiveness. Organisation has 
been included as a feature in a number of reporting 
tools such as CONSORT [8] and other checklists [6,12]. 
Often, this is included in the broader heading of set- 
ting'. However, it was important to make the checklist 
more specific and clear to the researchers. Few changes 
were made to this construct throughout the develop- 
ment process. 

Location 

Another characteristic of a setting is the location. The 
location construct was considered as important from the 
beginning of the checklist. In research proposals it is 
useful to know whether the location is in an area of 
deprivation, or if it is in an urban or rural location. This 
will ultimately affect the delivery of the intervention, 
which may not be generalizable in another location. 
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Table 2 The final checklist 

Organisation Is it clear which organisation(s) are involved in the study? 

Has the proposal considered: 

- How many study sites? How do these differ? 

- How big is the organisation? 

- What type of trust or authority is involved? e.g., Acute trust, ambulance trust, CCG, local authority, care trusts, 
mental health trusts, etc. 

- Other information on organisational type, e.g., foundation trust status, teaching/research beacon site, CQC or 
other ratings if relevant 

Location Is it clear where the organisation is located? 

Has the proposal considered: 

- What type of area it is, e.g., urban, rural, deprivation, etc. 

- What are the population demographics? 
Patient group Is it clear what the case mix of patients is? 

Has the proposal considered: 

- Who is receiving the intervention? 

- How many patients are being seen? 

- The characteristics of the patients, e.g., age group, health status, disease area. 
Workforce and Staffing Is it clear who are the main actors involved in service delivery? 

Has the proposal considered: 

- Is the total workforce clearly described? e.g., are the skill, grade, and profession mix stated? 

- How has the staff been selected? Are they already working at the organisation or are they new to the 
organisation? 

- What if any training will be needed for staff to deploy the intervention? 

Intervention Is there enough information about the service or intervention under exploration/examination? 

Has the proposal considered: 

- How is the intervention different from usual care? 

- What does it consist of? What does it look' like? 

- Is it clear when the intervention started and finished? 

- If there are information materials are these adequately described? 

- If there is a training programme, are there clear details? 

Other important contextual Are there any other contextual aspects which may affect the outcomes of this research and generalizability to 

information other sites? 

For instance, are there particular: 

- Policy initiatives related to the intervention (such as national or regional patient safety or improvement 
initiatives) 

- Particular issues at the study sites around exceptional leadership, for instance championing by nurse director 
or clinical team 

- Local features such as unusual geographical or service configuration 

- Events at the site, such as a critical safety incident or top team re-shuffle 



Patient group 

Patient group was included from the start of the devel- 
opment of the checklist. Most checklists which are used 
to report research emphasise the importance of having a 
good description of the participants. However, the word 
participant' can mean a range of groups who take part 
in a study. For the purpose of the checklist we made this 
specific to the patient group. Nevertheless, if the inter- 
vention is evaluating a workforce intervention, this is 



covered by the next construct (workforce and staffing). 
Therefore, the term patient group' was used as this ap- 
peared clearer to what is needed in research applications. 

Workforce and staffing 

Workforce and staffing were considered to be important 
from the outset of the development of the checklist. 
Elements of this construct have been seen in other 
checklists such as those used for patient safety [6]; 
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however, the wording has been adapted to suit the needs 
of this checklist. When originally forming the checklist, 
workforce and staffing were separate constructs. How- 
ever, when conducting the retrospective review, the two 
constructs were merged, as it was difficult to distinguish 
between the two. 

Intervention 

Originally, this construct was called service provided', as 
it was felt this would be the best wording to describe the 
active ingredient. However, there is often confusion 
about what the active ingredient is, especially when it re- 
lies on the context in which it is being delivered. For in- 
stance, an intervention such as care closer to home for 
children with complex conditions might build on some 
existing community nursing services but with some 
modification to skill mix or grade mix. As a result, this 
then overlaps with other constructs, such as workforce 
and staffing' and can appear unclear; this became par- 
ticularly apparent when completing the retrospective 
review. Therefore, for simplicity, this construct was 
changed to 'intervention' which would be clearer to the 
researchers. 

Other important contextual information 

When initially forming the checklist culture/leadership 
and costing' were each considered an essential construct. 
However, these were difficult to justify and it was not 
thought appropriate to represent them in a separate sin- 
gle construct, although they were of high importance. 
Therefore, a construct called 'Other Important Context- 
ual Information' was added to cover all other aspects, 
which would be missed by the other constructs. A num- 
ber of these elements were considered important, as 
highlighted in Taylor et al. [6], such as organisational 
and structural features of provider organisations (size, 
teaching status, and number of beds) with other facets, 
which may be more difficult to measure, such as levels 
of team working, cohesion, and leadership. However, 
when completing the retrospective review, these were re- 
moved from the checklist as it became apparent that it 
is particularly difficult to describe these in research pro- 
posals. Nonetheless, it is important to try and capture 
these important aspects in a research proposal. Thus, 
the other important contextual information' construct 
was developed to ensure the researchers have considered 
all characteristics of contextual information in their re- 
search proposal, particularly those that affect the 
generalizability to other sites. One of the main difficul- 
ties in health service research is generalising findings 
from one service to another, given the importance of 
context and local determinants of practice. Conse- 
quently, it was essential that these elements of a research 
proposal are clear and well described. Examples include 



any policy initiatives related to the intervention; issues at 
the study sites around exceptional leadership, for in- 
stance championing by nurse director or clinical team 
and local features such as unusual geographical or 
service configurations; whether there is participation in 
an improvement collaborative or Commissioning for 
Quality and Innovation initiative on particular clinical 
area; or a relevant pay-for-performance initiative for a 
clinical pathway. 

Discussion 

This paper reports on a small action-orientated mixed 
methods study which aimed to develop a checklist to be 
used to aid researchers when writing research proposals 
to submit to the NIHR HS&DR Programme. There are 
many constructs which would be useful to a researcher; 
however, some have changed shape or have been re- 
moved at various stages during the development of the 
checklist. The final six checklist constructs have a blur- 
ring between intervention and context, as debated in 
peer-reviewed literature [5]. The blurring was recognised 
as a difficulty for researchers, but it has not been pos- 
sible to keep the two separate. However, as the feedback 
confirms, having six clear constructs may make applying 
for research funding more straightforward for less expe- 
rienced applicants, so the blurring should be less of a 
concern. 

This study's main strength is that, to our knowledge, 
this is the first time someone has developed a checklist 
to aid the development of research proposals in health 
services and delivery research. There are also several 
limitations that need to be acknowledged. Firstly, in the 
retrospective review of research proposals we only in- 
cluded a small selection of research proposals. Future re- 
search could look at a different and larger selection of 
research proposals. Next, only two people conducted the 
retrospective review. Therefore, there is a possibility that 
someone with different expertise would have collected 
different information for the checklist or would have 
had a different view on the checklist. Lastly, it is import- 
ant to highlight that only some initial small-scale work 
was completed with a sample of funded researchers to 
test usefulness, acceptability, and face validity of the 
constructs and checklist. Future work could include 
more rigorous validation of the checklist. 

Initial positive feedback from researchers indicated 
that this checklist may be a useful tool to help address 
the problem of not being able to fund important re- 
search proposals. The work presented in this paper 
should be considered the first step in a longer-term 
process of subsequent research. The checklist was devel- 
oped essentially as an in-house, pragmatic tool for the 
NIHR HS&DR Programme and further research is 
needed to strengthen the checklist. It would be useful to 
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test the checklist constructs with a wider sample of 
stakeholders using techniques such as consensus build- 
ing or a Delphi model 

Conclusions 

There is a body of literature which discusses interven- 
tion, context, and the use of checklists. However, these 
checklists are not always relevant to complex, staff- 
based organisational research in healthcare; existing 
checklists do not have enough focus on areas relevant to 
complex health service interventions or consider re- 
search applications. We have developed a checklist, with 
a focus on complex health services and delivery inter- 
ventions and context. Small-scale iterative testing sug- 
gested it was acceptable and useful for researchers. 
Further validation is needed to demonstrate relevance to 
a wider range of researchers and funding bodies. 
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